Skip to content

feat: add test_helpers module (error_utils, test_utils) behind function_runner flag#2378

Closed
naor-starkware wants to merge 8 commits intonaor/refactor/rename-feature-flagfrom
naor/feat/add_test_helpers
Closed

feat: add test_helpers module (error_utils, test_utils) behind function_runner flag#2378
naor-starkware wants to merge 8 commits intonaor/refactor/rename-feature-flagfrom
naor/feat/add_test_helpers

Conversation

@naor-starkware
Copy link
Copy Markdown
Collaborator

@naor-starkware naor-starkware commented Mar 29, 2026

TITLE

Description

Description of the pull request changes and motivation.

Checklist

  • Linked to Github Issue
  • Unit tests added
  • Integration tests added.
  • This change requires new documentation.
    • Documentation has been added/updated.
    • CHANGELOG has been updated.

This change is Reviewable

Copy link
Copy Markdown
Collaborator Author

naor-starkware commented Mar 29, 2026

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 29, 2026

Codecov Report

❌ Patch coverage is 94.80519% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.06%. Comparing base (687552e) to head (3e8d6ce).

Files with missing lines Patch % Lines
vm/src/test_helpers/test_utils.rs 84.00% 8 Missing ⚠️
vm/src/test_helpers/error_utils.rs 97.79% 4 Missing ⚠️
Additional details and impacted files
@@                          Coverage Diff                          @@
##           naor/refactor/rename-feature-flag    #2378      +/-   ##
=====================================================================
- Coverage                              96.07%   96.06%   -0.01%     
=====================================================================
  Files                                    105      107       +2     
  Lines                                  37737    37968     +231     
=====================================================================
+ Hits                                   36254    36475     +221     
- Misses                                  1483     1493      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 29, 2026

Benchmark Results for unmodified programs 🚀

Command Mean [s] Min [s] Max [s] Relative
base big_factorial 2.133 ± 0.026 2.104 2.174 1.01 ± 0.01
head big_factorial 2.115 ± 0.011 2.102 2.138 1.00
Command Mean [s] Min [s] Max [s] Relative
base big_fibonacci 2.054 ± 0.015 2.042 2.090 1.00
head big_fibonacci 2.057 ± 0.023 2.040 2.116 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base blake2s_integration_benchmark 7.462 ± 0.094 7.358 7.633 1.00 ± 0.02
head blake2s_integration_benchmark 7.448 ± 0.101 7.371 7.637 1.00
Command Mean [s] Min [s] Max [s] Relative
base compare_arrays_200000 2.190 ± 0.017 2.163 2.215 1.00
head compare_arrays_200000 2.194 ± 0.017 2.173 2.222 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base dict_integration_benchmark 1.433 ± 0.010 1.420 1.448 1.00 ± 0.01
head dict_integration_benchmark 1.428 ± 0.014 1.416 1.449 1.00
Command Mean [s] Min [s] Max [s] Relative
base field_arithmetic_get_square_benchmark 1.227 ± 0.006 1.223 1.241 1.00
head field_arithmetic_get_square_benchmark 1.229 ± 0.009 1.221 1.248 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base integration_builtins 7.510 ± 0.034 7.478 7.575 1.00
head integration_builtins 7.666 ± 0.539 7.455 9.200 1.02 ± 0.07
Command Mean [s] Min [s] Max [s] Relative
base keccak_integration_benchmark 7.615 ± 0.024 7.562 7.652 1.00 ± 0.00
head keccak_integration_benchmark 7.578 ± 0.026 7.528 7.614 1.00
Command Mean [s] Min [s] Max [s] Relative
base linear_search 2.153 ± 0.014 2.142 2.192 1.00
head linear_search 2.162 ± 0.017 2.145 2.205 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base math_cmp_and_pow_integration_benchmark 1.494 ± 0.004 1.486 1.497 1.00
head math_cmp_and_pow_integration_benchmark 1.495 ± 0.002 1.492 1.499 1.00 ± 0.00
Command Mean [s] Min [s] Max [s] Relative
base math_integration_benchmark 1.464 ± 0.004 1.457 1.473 1.01 ± 0.00
head math_integration_benchmark 1.450 ± 0.004 1.445 1.455 1.00
Command Mean [s] Min [s] Max [s] Relative
base memory_integration_benchmark 1.224 ± 0.015 1.213 1.261 1.00 ± 0.01
head memory_integration_benchmark 1.218 ± 0.006 1.212 1.233 1.00
Command Mean [s] Min [s] Max [s] Relative
base operations_with_data_structures_benchmarks 1.541 ± 0.004 1.536 1.547 1.00 ± 0.00
head operations_with_data_structures_benchmarks 1.536 ± 0.005 1.530 1.546 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
base pedersen 534.2 ± 5.0 530.7 546.8 1.00 ± 0.01
head pedersen 532.2 ± 2.1 530.5 537.8 1.00
Command Mean [ms] Min [ms] Max [ms] Relative
base poseidon_integration_benchmark 624.6 ± 3.0 621.3 631.0 1.00 ± 0.01
head poseidon_integration_benchmark 624.6 ± 2.8 621.4 631.8 1.00
Command Mean [s] Min [s] Max [s] Relative
base secp_integration_benchmark 1.812 ± 0.005 1.805 1.821 1.00
head secp_integration_benchmark 1.818 ± 0.009 1.805 1.837 1.00 ± 0.01
Command Mean [ms] Min [ms] Max [ms] Relative
base set_integration_benchmark 665.9 ± 1.9 663.5 669.6 1.00
head set_integration_benchmark 669.2 ± 7.8 664.1 688.0 1.00 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base uint256_integration_benchmark 4.256 ± 0.017 4.230 4.280 1.00
head uint256_integration_benchmark 4.272 ± 0.077 4.234 4.491 1.00 ± 0.02

@naor-starkware naor-starkware force-pushed the naor/feat/add_test_helpers branch from 6ec5328 to ce3aeb1 Compare March 29, 2026 14:19
@naor-starkware naor-starkware marked this pull request as ready for review March 30, 2026 10:35
Copy link
Copy Markdown
Collaborator

@OmriEshhar1 OmriEshhar1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@OmriEshhar1 reviewed 7 files and all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on naor-starkware).

naor-starkware and others added 8 commits April 6, 2026 12:50
…on_runner flag

- Create vm/src/test_helpers/ with error_utils.rs and test_utils.rs
- Move from cairo_test_suite/ (fix filename typo: utlis → utils)
- Fix crate:: import paths (were cairo_vm:: when outside the crate)
- Fix $crate in macro_export macro (clippy::crate_in_macro_def)
- Simplify load_cairo_program! path using with_file_name()
- Gate module behind function_runner feature in lib.rs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ram! and error_utils checkers

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add AlwaysFailConversion helper + 2 tests for assert_mr_eq! unwrap_or_else
  panic branch (no-message and message variants)
- Allow clippy::result_large_err on hint_err test helper

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… noise

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…y function name error

#[macro_export] macros containing closures (|x| ...) cause llvm-cov to
emit a "function name is empty" error. Replaced unwrap_or_else(|e| panic!(...))
with match expressions to eliminate closures from macro expansions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Follow-up to dropping the function_runner feature flag.
Gate test_helpers module and function_runner module under test_utils,
and update the doc comment in function_runner.rs accordingly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@naor-starkware naor-starkware force-pushed the naor/feat/add_test_helpers branch from e97f3a8 to 3e8d6ce Compare April 6, 2026 09:53
@naor-starkware naor-starkware deleted the branch naor/refactor/rename-feature-flag April 6, 2026 17:39
naor-starkware added a commit that referenced this pull request Apr 6, 2026
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Collaborator

@YairVaknin-starkware YairVaknin-starkware left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@YairVaknin-starkware reviewed 8 files and all commit messages, and made 5 comments.
Reviewable status: all files reviewed, 5 unresolved discussions.


CHANGELOG.md line 17 at r2 (raw file):

* refactor: add `CairoFunctionRunner` type alias for `CairoRunner` under the `test_utils` feature flag [#2377](https://github.com/starkware-libs/cairo-vm/pull/2377)
* refactor: add `function_runner` feature flag and `CairoFunctionRunner` type alias for `CairoRunner` [#2377](https://github.com/starkware-libs/cairo-vm/pull/2377)
* feat: add `test_helpers` module (`error_utils`, `test_utils`) with `assert_mr_eq!`, `load_cairo_program!` macros and `expect_*` error checkers, behind `test_utils` feature flag [#2378](https://github.com/starkware-libs/cairo-vm/pull/2378)

fix

Code quote:

* refactor: add `function_runner` feature flag and `CairoFunctionRunner` type alias for `CairoRunner` [#2377](https://github.com/starkware-libs/cairo-vm/pull/2377)
* feat: add `test_helpers` module (`error_utils`, `test_utils`) with `assert_mr_eq!`, `load_cairo_program!` macros and `expect_*` error checkers, behind `test_utils` feature flag [#2378](https://github.com/starkware-libs/cairo-vm/pull/2378)

vm/src/test_helpers/error_utils.rs line 44 at r2 (raw file):

/// Type alias for check functions that validate test results.
pub type VmCheck<T> = fn(&std::result::Result<T, CairoRunError>);

can just be Result, right? pls looks for similar instances where you can shorten.

Code quote:

std::result::Result

vm/src/test_helpers/error_utils.rs line 62 at r2 (raw file):

}

/// Asserts that the result is `HintError::AssertNotEqualFail`.

This funcs have very repetitive boiler plate. Pls extract to a single func which these func will invoke that will get the res and the predicate u check.

Code quote:

/// Asserts that the result is `HintError::AssertNotEqualFail`.

vm/src/test_helpers/error_utils.rs line 220 at r2 (raw file):

    }

    /// `expect_hint_assert_not_zero` does not panic on `HintError::AssertNotZero`.

these error tests could be parameterized using rtest to reduce alot of repetitive boilerplate.

Code quote:

/// `expect_hint_assert_not_zero` does not panic on `HintError::AssertNotZero`.

vm/src/test_helpers/test_utils.rs line 48 at r2 (raw file):

            Ok(v) => v,
            Err(e) => panic!("conversion to MaybeRelocatable failed: {e:?}"),
        };

pls factor out the conversion logic of both cases into a single func which also enforces coercion into MaybeRelocatable (currently it will work for any right that is able to be try_into'd left's type).

Code quote:

        let right_mr = match ($right).try_into() {
            Ok(v) => v,
            Err(e) => panic!("conversion to MaybeRelocatable failed: {e:?}"),
        };

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants